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Limitations of Traditional Cooling Methods 


e Traditional air cooling is very inefficient and capital intensive. 
e Multiple heat transfer processes (2"4 Law) 
e Mixing of hot and cold airstreams 
e Fans, pumps, blowers and compressors consume power. 
e Reliance on air as a heat transfer medium means high AT at heat sink. 


Chiller Cooling 
Tower HEAT OUT 
155-220W 
55-120W for 
Cooling 
27-58W 
10W Fan Fan 1W Pump Compressor 1W Pump DER 
+ water 
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Limitations of Traditional Cooling Methods 


e Traditional water or refrigerant cooling offers efficiency and density but is 
expensive and complex [1]. 


e Power density is greatly increased by elimination of airflow paths. 


e Energy efficiency is high because liquid and device temperatures are tightly coupled. This 
enables “warm” cooling techniques. 


e High temperature liquid stream permits distribution and utilization of waste heat. 


e Much of the hardware is duplicated. 


e Inability to control facility water chemistry, 
pressure, etc. mandates a separate water 
loop for the rack and an additional heat 
transfer process in the coolant distribution 


unit (CDU). 

e It is costly to design and manufacture leak- 
proof networks of pumps, valves, hoses, Water-cooled 
manifolds, cold plates, fittings, etc. IBM BladeCenter HS22 


IBM - Zurich 
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Limitations of Traditional Cooling Methods 


j 


IBM Power 775 Supercomputer 
Rack (>200kW) 


Courtesy of International BusiNess Machines Corporation. 
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upercomputer 


Partial list of liquid cooling hardware in 
one P775 rack [2,3] 


Node Level 
7096 |thermal interfaces by grease or gap pad 


48 |water-cooled DIMM cold rails 
node module hose sets 


Inodemodulehosesets | 


| 12 |32"x21" node DIMM/VTM cold plate 

| 12 |32"x12" node hub cold plate 

| 8 [PSU cold plates, large aluminum with Cu D-tubes, and QDs | 
1"x2" L-shaped welded SS manifold assemblies, 19 ports 

radiator core for drive bay 

Rack Level 


| 4 |90kw water conditioning unit, $23,000 each 
rear door heat exchanger, $12,000 
| 1 [coolant fill/purge tool $45,000 
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Limitations of Traditional Cooling Methods 


e “Free” air cooling is increasingly efficient 
but density limited. 


e Commodity CPUs easier to cool. 


— CPU case temperature specifications have been rising. 
Tease=8°C not unusual for 130W CPU. 


All Year 


Class A2 


— Package and heat sink thermal performance has Dry Bulb < ET S 
eS Dew Point < 21°C a 
e Allowed Emergence of 2011 ASHRAE facility a Jeane 
classifications A3 and A4, [4]. The Green Grid 


— A4-compliant equipment can use air as hot as 45°C. “or 


— Enables free cooling just about anywhere and year 
round. 


— Fan power is low much of the year. 


e PUEs <1.05 (excluding chassis fan power) are 


| . : Class A3 
possible even in warm climates [5]. 


Dry Bulb < 40°C 5 
e Increased density comes at the expense of Dew Point < 24°C 
efficiency. 


Le 0 days 
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Limitations of Traditional Cooling Methods 


e Traditional 2-phase immersion cooling offers efficiency 
and density but is expensive and complex. 


Military Enclosure, active 
e Anelegant and well-established way to capture ALL heat spray (photo courtesy of 


generated by a complex and very dense electronics Spraycool) 
assembly 


e Eliminates a complex network of plumbing. 


e tis inherently costly and complex to build hermetic 
electronic enclosures particularly when many conductors 
must penetrate that enclosure. 

e Hermetic connectors are expensive. 
e Charging/degassing complicated. 


e Servicing is complicated if one has to open a pressure vessel 
to access equipment. 


Airborne radar transmitter, 
Raytheon, -circa 1967 
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Open Bath Immersion Cooling Power/lO a 
Concept Conduit - 


e Overview [6] 


e Servers are placed side-by-side in a lidded Condenser 
bath of dielectric fluid. 


e Devices cause fluid to boil. 


Lid 


e Rising vapor condenses on a condenser 


cooled by facility water. Vapor, T; 


e Servers plug into an immersed backplane. 


e Power and IO enter/exit through a conduit 


; ae Nod 
that terminates below the liquid level. i 


e Servers can be hot-swapped and leave the 


bath dry. Boiling Two 
e The bath is “semi-open” as it is at Liquid, 

atmospheric pressure. Bath breathes Ts racy 

through a trap (not shown). rank 


Backplane 
Bath Cross Section 
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Open Bath Immersion Cooling Concept 


e Advantages 


e Allserver- and most rack-level cooling hardware are 
eliminated. 


e Increased packaging density can increase 
communication bandwidth [7] 


e Thermal efficiency is high: 


e The fluid boiling and condensation temperatures 
are the same. There is no advection. 

e No Secondary or ternary thermal interfaces < 
between the silicon and fluid 

e Fluid boiled from an optimized surface that 
produces heat transfer coefficients >100,000 
W/m2-K. 


e All devices are kept at the same constant temperature. 


Il 


e Fluid losses occur... i 
e not at hundreds of intractable sites from a high pressure system... 


e but at 2 sites at which the physics of fluid loss are defined and easily 
controlled and from a system near atmospheric pressure. 


DatacenterDynamics 


Le Palais des Congres de Paris, 5 Jun, 2012 


Energy Efficiency 
e Three Heat Transfer Processes: 
e CPU Junction-to-Fluid, AT; - Dictated by CPU package and boiling technology 
e Fluid-to-water, AT; - Dictated by condenser design/size, water flow, etc. 


e Water-to-ambient, AT, - Dictated by tower fan, volume, etc. 


e Two Power Inputs 
e Fan 


Boiling Fluid, T; 
e Pump 
Wean 


Saturated Vapor, T; 
CPU Junction, T, 


Ta] 


Facility Water 


Facil 
acility 
Wall Pump 


Dry Tower 
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Energy Efficiency 
e CPU Junction-to-Fluid Performance, AT; 


e Boiling heat transfer enhanced with a porous copper Boiling Enhancement Coating (BEC) 
applied to the lid of the microprocessor. 


e 15X increase in boiling heat transfer coefficient 
e 70% increase in dryout heat flux 


Ts 


Interface © 


15 T 
-© BEC-enhanced £ 
va 4 Plain Copper et Junction, T; 
a 10 rs 
: . 
= 5 a 
I Boiling performance of 
0 BEC vs. plain surface 
0 10 20 30 40 Cross section and SEM of BEC 
Q" [W/cm?] 
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Energy Efficiency 


e Production lid has no BEC and is often too thin 
to adequately spread heat. 


e BEC coupon can be attached with solder. 


e Resultant junction-to-fluid thermal resistance, 
Ri depends upon 
e Chip core size, silicon thickness 


e Type of thermal interface between chip 
and lid 


BEC coupon 
soldered onto 
AMD Processor 


Junction-to-Fluid Thermal Resistances Measured with Real Devices (Based on “On-Chip” Sensors?) 


Device | Die Area [cm] Thermal Interface Est. Power [W] Measured R; [°C/W] 


IBM PowerPC® 970FX | 066 l metal alloy | 90 
tFrom [8] Opteron is a registered trademark of AMD 
Power 6 is a modeled value [9] and the cores are ~0.5cm? POWER6 and PowerPC are registered trademarks of IBM 
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Energy Efficiency 


e Fluid-to-water Performance, AT;, 
e Best condenser design depends upon the system power. 


e Conventional radiators work well for small systems (vapor condenses on air side). 


e Enhanced tube bundles more appropriate for large scale systems (10s of kW). 


e Resultant R,,, depends upon design, size and water flow. 


CLO . 
OKO : 
OSO | 


© 
© 
O 


30kW condenser design based 
on enhanced copper tubing of 


the type used in large chillers. © O 
Core volume ~15 liters and log © © © 
mean temperature difference is 

10C OO 
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5 
OLS 
000 


Passes/cireuit | 8 
PassLlem] O 
Total water flow [liter/min] | 60 
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e Actual Immersion-Cooled Server Cluster [10] Video YouTube Channel petumal 
e Based on 4 Supermicro MBD-H8DMT-F-B boards. 
e 8 AMD Opteron™ 83VS Processors : 115W each at full load, total power 1.2kW 


-—8 Ethernet to Network 
AC Server Power 


TOP VIEW 


Condenser 


Oa Nc 0 
FRONT VIEW 
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120x360mm 
Radiator 


DC Fan 
Power 


Water 
Reservoir 


AC Pump 
Power 


Mass Flow D. _ 
Before Modification 7 


SIDE VIEW 
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Energy Efficiency 


Simple PUE = (Fans + Pump + Server) / Server 


Server Cluster perimien Data — Fluid 1 boiling point 61°C 


Immersion- 
ran | rump | n | [o | Tm [erin ue cooled server 
ee | 


Med 


Hi | Hi 
Lo 


Server Cluster Projected Data — Fluid 2 boiling point 76°C 


EE eee ee 


Experimental Data 


Projected 
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Energy Efficiency 


100 
90 
Ts =85°C 
80 case 
70 Temperature at which water might be 
— zy economically-viable for desalination and 
oe 60 adsorption refrigeration [11] 
z Maximum water temperature entering a facility rated 
50 a “Liquid Cooling Class 3” as proposed by High 
Performance Computing Working Group (HPCWG) [12] 
40 | 
m Server Cluster Projected 
30 Data — Fluid 2 boiling 
point 76°C 
20 


1:5 3.4 
Liters/min-kW 
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Power Density 


We could not find computing hardware sufficiently dense to demonstrate the 
power density capability of passive 2-phase immersion. A 4 kW node simulator 
was therefore built. 


| em, 4kW blade simulator 
(17x20cm, twenty 200W CPU simulators) 


Note: bath, condenser 
and headspace each 


CPU Simulator Detail occupy about 1 liter. 


(19x19mm ceramic heater, 
“Sink” thermocouple in 30x30x3mm BEC-enhanced copper boiler) 
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Power Density Data 


Acquisition 


TH20,in Tho, out 
poa 
= à 


mass flow 


individual “sink” 
temperatures 


Power 


fluid saturation 
temperature 


Liquid 
Vapor 


Schematic of the test setup for experiments conducted with 4kW node 
simulator. 3 different fluids were used. 
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100 5 Water temperatures and 
pe FLUID 2 flow rates from 
80 4 — with 4kW node simulator. 
Fluid Saturation Temp = 
we 
70 E 
a = 
> 60 3 Š 
p Z 
= 50 © 
(40) = 
Dim ; 
S Ç z 
39 ___ i 
ri : 
20 1 
[ i a Tiom [°C] 
| 0 = Toul Al 
0 0 m M/Q [kg/min-kW] 


Power =Q [kW] 


Projected junction temperature, T; prop based on a 150W CPU and average junction-to-fluid thermal resistance of 0.060°C/W. 
In an optimized run, the water flow rate is adjusted so that the condenser is just able to condense the vapor being generated. 
* Indicates a non-optimized run. 

+ Indicates projected data — tap water would not get hot enough to perform this experiment. 
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Power Density 


e System level capability 


40 


35 


N 
[ea 


z 
=x 
re 
2 
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m. 
UT 


m Cha 


E Condenser 
@ Headspace 
4 @ Node — 


ssis/Rack 


4x air-cooled node 


density 


P775Rack Typical 1U 4kW Demo 
Chassis 


More 
Realistic 


Comparison of 
volumetric rack and 


|| chassis power 


densities with system 
level capabilities of 


|| immersion 
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Power Density 
e Floorspace 


0.3 


0.25 


0.05 


& Work/Walk 
m Plena 
@ Rack/Bath 


P775 Rack 
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Air-Cooled 


More Realistic 


Projection: 80kW @ 4x air- 
cooled node density 
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Simplicity 


e Elimination of duplicitous node level cooling hardware 
e Atmospheric pressure operation 
e Passive heat transfer process — no pumps, controls, etc. 


e Inexpensive subsystems functioning at bath scale: 


Passive organic filter (activated carbon) 


Passive desiccant to remove moisture at startup 


— Vent system 
e 2 Pressure Switches — sense +1cm H,O 
* Solenoid valve — vent system as needed 
e 2 Check valves — backup for solenoid 
e Bellows —Accommodate small changes in vapor height 
e Secondary condenser or “trap” 


Hot swap condenser Hot swap experiments show 
* Simple condenser active only when performing hot swaps very low fluid losses. These 
must be validated on a large 

scale. 
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Vent Check Valve 
Vent Solenoid Valve 


š Vent Switch 
Components in 2-phase 
immersion cooling bath system Vacuum Break Switch 
Hot Swap 
Water 


Pump 


Air Filter 
Vacuum Break Check Valve 
Vacuum Break Solenoid Valve 


10/Power 


Conduit Bellows 


Vent Condenser 


ai hot swap radiator + Fan 


| Desiccant 


Moisture Indicator 


Lid + Gaket 


Hot Swap Condenser z 


Primary Condenser Flow Switch 


a = Falih 


water 


BE Switch Tank Low 
SA (STL) 


Organic Filtration 


Tank — Welded metal construction with no gasketed 
penetrations from condenser downward. 
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~10 


l | 


——— 


Commodity enhanced tubes like those 
used in low-pressure chiller 
condensers are ideal for use in reflux 


/) 


condenser. ala 
or 
thermostatic metering 

Purge Valve 


valves can modulate 
facility water flow to 
maintain a certain 
vapor height in the 
bath. 


Pressure 
Switch 


Secondary condenser is active 
only when bellows capacity is 
z near and system must vent air. 
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Activated carbon adsorbents will 
remove organic fluid 
contaminants that could foul the 
boiling surface. 


Bellows made from food packaging film can 
accommodate the air/vapor mi t would 
otherwise be v 
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Future Work 


e Out of the Lab, into the Field 
e Demonstrate Feasibility and Cost of Ownership 


e Demonstrations pending in 3 countries 
e 75kW Immersion Bath 
e Initially heaters only, 
e To test power density and condenser performance at larger scale 
e  25kW Compute Cluster 
s 144 Processors 
e Demonstrate OBI on larger scale 
e 25kW Compute Cluster 
e Test limits for waste heat recovery 
e  25kW Compute Cluster 
e — Conversion of existing super computer hardware 
e Test air-cooled equivalent alongside 
e ?kW Compute Cluster 
e Demonstrate OBI in hot, humid climate with high population density 
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Conclusions 


e Open Bath Immersion Cooling Offers 


— Density 
e Long term node-level power density capability 4kW/liter 
e Near term floor space density similar to water-cooled supercomputers 


— Energy Efficiency 
e Potential for PUET1.02 with 55°C ambient 
e Potential for PUE <1.01 with 37°C ambient 
e Efficient heat capture for re-use 
— Simplicity 
e Complete elimination of node level cooling hardware 
e Passive heat transfer to facility water 
e Atmospheric pressure operation 
e Simple, dry hot swappability 
e Simple controls applied not at node but bath level 
e Silent elimination of air cooling infrastructure 


e Large Scale Demonstrations Pending 
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